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ABSTRACT 



The Schools and Staffing Survey (SASS) has made it possible 
to develop comprehensive profiles of the nation's public and private schools 
and teachers and principals and to study issues of teacher supply and demand. 
The planned switch to a 5-year cycle of data collection provides a good time 
to re-examine the survey's design. This paper focuses on an aspect of design, 
survey depth. It addresses questions of the level at which estimates should 
be provided, from whom data should be collected, and how much data can 
reasonably be collected from each respondent. The paper starts with a 
description of the context within which choices can be made about survey 
depth. It examines the purposes of the SASS, identifies emerging policy 
issues, and describes users and uses of the data. This exploration of survey 
depth leads to a conclusion that, given the current focus on measuring 
quality at the classroom level, the SASS needs to reach more extensively into 
the classroom. This means gathering more information on teacher training and 
professional development. State- and affiliation- level estimates should be 
continued, as should the current oversampling of schools with large American 
Indian/Alaskan Native population, but no additional types of respondents 
should be added. The increased response burden will fall mainly on teachers, 
and modifications to keep the burden on teachers realistic may be required. 
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Foreword 



Each year a large number of written documents are generated by NCES staff and 
individuals commissioned by NCES which provide preliminary analyses of survey results and 
address technical, methodological, and evaluation issues. Even though they are not formally 
published, these documents reflect a tremendous amount of unique expertise, knowledge, and 
experience. 

The Working Paper Series was created in order to preserve the information contained 
in these documents and to promote the sharing of valuable work experience and knowledge. 
However, these documents were prepared under different formats and did not undergo vigorous 
NCES publication review and editing prior to their inclusion in the series. Consequently, we 
encourage users of the series to consult the individual authors for citations. 

To receive information about submitting manuscripts or obtaining copies of the series, 
please contact Suellen Mauchamer at (202) 219-1828 or U.S. Department of Education, Office 
of Educational Research and Improvement, National Center for Education Statistics, 555 New 
Jersey Ave., N.W., Room 400, Washington, D.C. 20208-5652. 
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INTRODUCTION 



The Schools and Staffing Survey (SASS), first fielded in 1987-88, represented a major 
redesign of the NCES elementary and secondary data collection, and it has proved to be a 
successful one. Thoughtfully designed and competently executed, SASS’s linked surveys have 
made it possible to develop comprehensive profiles of the nation’s public and private schools and 
teachers and principals and to examine supply and demand issues more thoroughly than ever 
before. Successive administrations in 1990-91 and 1993-94 brought improvements in sample and 
questionnaire design and additional information on timely topics. The switch to a 5-year cycle 
of data collection provides some time to reexamine the survey’s design and consider changes to 
enhance the relevance and quality of the SASS data collection effort. 

This paper focuses on one aspect of the survey’s design, survey depth. It addresses three major 
questions: 

• At what level at should estimates be provided? Are state- and affiliation-level 
estimates useful enough to justify the large sample size they require? Are the 
oversampled groups of schools and teachers still important to single out? 

• From whom should data be collected? Should any new categories of respondents 
be added to the survey? 

• How much data can reasonably be collected from each respondent? What options 
are available to expand data collection without overburdening respondents? 



The paper starts with a description of the context within which choices about survey depth 
must be made. It examines the purpose of SASS, identifies some important emerging policy 
issues and their implications for SASS, and describes the users and uses of the data. The paper 
then takes each set of questions in turn, discussing the issues they raise and making specific 
recommendations on how to deal with them. 



CONTEXT FOR CONSIDERING SURVEY DEPTH 

To make appropriate decisions about survey depth, we must first consider the purpose of the 
survey and who uses (or might use) it to what end. Although the future scope of SASS is 
currently being debated, some assumptions were necessary in order to make recommendations 
about survey depth. For the purposes of this paper, I have assumed that the general goals and 
structure of the survey will remain intact, but have also taken into account emerging policy issues 
that might appropriately be monitored through SASS. Thus, although the focus of this paper is 
survey depth, it unavoidably strays into the realm of survey scope. 



Purpose of the Schools and Staffing Survey 

SASS was designed to inform Congress, the Department of Education and other federal 
agencies, state education agencies, local districts, educational associations, and the larger 
education community on four major topics: 1) teacher supply and demand; 2) school conditions 
and programs; 3) the characteristics of the elementary and secondary teaching force; and 4) the 



characteristics of principals. 1 In the mid-1980s, when the first SASS was designed, many analysts 
were predicting a serious impending teacher shortage, particularly in math and science. An 
important purpose of SASS, therefore, was to monitor the flow of teachers around the country 
and in and out of the teaching work force in ways that earlier elementary and secondary data 
collections were unable to do. 

After more than a decade of concentrated attention to school reform at the federal, state, and 
local levels, these four general topics remain as relevant as ever. The focus of some of the 
questions related to them has changed, however. For example, when the predicted teacher 
shortages did not materialize, researchers and policymakers shifted the emphasis of their work 
from the absolute number of teachers to more complex issues such as the supply of teachers in 
specific fields, the distribution of qualified teachers across various types of schools and subject 
areas using various definitions of “qualified,” the match between qualifications and assignments 
as indicators of shortages in various fields, difficulties filling new positions, the adequacy of the 
supply of minority teachers, and the relationship of school and teacher characteristics to attrition. 

Although teacher shortages may not exist in the aggregate now, monitoring supply and 
demand remains a worthy goal, because shortages could still become a problem in the future. In 
1993-94, about one-quarter of all newly hired teachers were not teaching the year before. 2 Thus, 
the size of the pool of potential teachers is uncertain and subject to the influence of many factors 
such as teacher salaries, the attractiveness of alternative careers, and the general state of the 
economy, to name a few. 

Like supply and demand, school conditions and programs and the characteristics of the school 
work force still command attention, although the focus of some of the questions has changed in 
these areas as well. SASS has been updated to reflect this. Compared with previous 
administrations of SASS, the 1993-94 SASS has more questions on teacher education and 
certification, professional development, the influence of various participants in school decision 
making, and the variety of programs and services schools offer. 

In summary, SASS has been well structured to capture information on enduring issues and has 
been modified appropriately to reflect changing policy concerns through the early 1990s. Now, 
however, after more than 10 years of reform initiatives, it is important to take a hard look at the 
information being gathered in light of the need to monitor the implementation and impact of these 
initiatives and to try to anticipate what people will want to know about schools and staffing 5 
years from now when the data from the next SASS administration are released. No one would 
argue that an ongoing national survey monitoring conditions in the schools should be redesigned 
to follow every educational fad. Nevertheless, policies and priorities do change over time, and 
it is important for SASS to remain relevant and timely. 



1 U.S. Department of Education, National Center for Education Statistics, Programs and Plans (Washington, D.C.: 
1995), 9-10. 

2 U.S. Department of Education, National Center for Education Statistics, Schools and Staffing in the United States: 
A Statistical Profile, 1993-94 (Washington, D.C.: forthcoming). 



Emerging Issues 

As we head toward the 21st century, some of the major issues that education researchers and 
policymakers are focusing their attention on are the following: 

• What students are taught (curriculum frameworks, course content, graduation 
requirements) 

• How students are taught (instructional practices, use of technology) 

• How their progress is evaluated (tests, portfolios) 

• Who is teaching and how they prepare for this responsibility (sources of teachers, 
minority teachers, content of teacher education, professional development) 

• How students are prepared for school (preschool education, early childhood experiences) 
and how they are assisted in making the transition from high school to further education 
or work (school-to-work transition programs, tech prep programs) 

• How schools are organized and managed (site-based management, decision-making 
processes, choice, vouchers, magnet schools, charter schools, networks) 

Implications for SASS 

There is much we want to learn about these reforms, but it is important to pause and think 
about what SASS can and cannot do well given its overall size and design. First, SASS should 
only ask about things that are readily measured and that have a readily understandable 
definition. At the recent NCES-sponsored conference on possible future directions for NCES, 
Jennings and Stark note this as a barrier to collecting information on school reform. They suggest 
establishing an advisory committee made up of individuals with widely varying views on 
education reform to help determine what to study and develop common definitions. 3 However, 
even if common definitions could be agreed upon through such a process, we certainly cannot 
assume that these definitions will mean the same thing to all the individuals answering questions 
about them. This is especially true for reforms that are new and not clearly defined. 

For example, the 1993-94 SASS asked schools if they offered a “tech prep” program. Tech 
prep was defined as “vocational-technical instruction in the last two years of high school designed 
to prepare students for two years of vocational instruction at the postsecondary level.” With this 
definition, almost any school with a vocational program could respond affirmatively to this 
question, and, in fact, 56 percent of public schools with 12th grades reported that they had such 
a program. 4 “Tech prep” has now come to mean something much more specific, with the critical 
defining feature being some type of formal articulation between secondary and postsecondary 
programs. Consequently, not much is learned from this question. This example also illustrates the 
dangers of trying to measure something too soon, before there is a common understanding of 
what it is. If a reworded tech prep question is included in the next SASS, the percentage of 



3 John F. Jennings and Diane Stark, “Tracking Educational Reform: Implications for Collecting National Data 
Through 2010” (paper prepared for the Conference on Future Directions of NCES Data Collection: Some Possible 
Directions, November 1995), 4-5. 

4 NCES, Schools and Staffing in the United States: A Statistical Profile, 1993-94. 



schools with tech prep programs will probably appear to go down, which would not be an 
accurate depiction of what is going on. NCES must guard against adding questions on “hot 
topics” that do not yet have commonly understood or easily communicated definitions. 

SASS can only measure things that are widespread. National surveys are not useful for 
providing information on reforms that involve relatively small numbers of schools or teachers, 
as new structures and practices often do. For example, some schools are supporting each other 
in the reform process by establishing networks of schools with common goals, such as the 
Coalition for Essential Schools. In a nationally representative sample, you will not pick up 
enough schools to say anything useful about schools that belong to this network or to compare 
them with schools that belong to other networks. For now, if you want to study these networks, 
the best approach would be to start with the list of schools that belong. Similarly, we might one 
day want to track the movement of National Board Certified Teachers. 5 At present, however, 
there are fewer than 1,000 such teachers, and you might not pick up any of them in a national 
survey, and certainly not enough to make any comparisons to other teachers. 

Of the different types of reforms and changes going on in schools and the teaching profession 
now, some are more suitable for including in SASS than others. It is relatively easy to ask who, 
what, or how many. Much harder to answer are “how” questions. Thus, it is fairly easy to ask 
about professional development activities (how many workshops of a certain duration a teacher 
attended, for example, and in what subject areas) or teacher education (measured in terms of 
courses, majors, minors, or degrees). It is also relatively easy to measure how many students 
participated in certain programs (like tech prep), or how many plan to apply to college. It is also 
feasible to ask about the existence of certain policies than have readily understandable meanings 
(for example, choice programs or vouchers). 

Much harder to measure are organizational management and issues such as how decisions get 
made. For these, national data collections such as SASS are less useful. In 1993-94, for example, 
teachers and principals did not agree on how much influence teachers have on certain types of 
decisions. This is worth knowing, but also does not give a clear picture of what was going on in 
schools. Because organizational processes are so complex, I therefore disagree with Baker that 
SASS would be very useful for studying these processes. 6 Smaller studies using interviews would 
be more appropriate. A further complication is that the teachers in SASS are not representative 
of the teachers in their school. Their perceptions about how their school works could differ quite 
markedly from the perceptions of their colleagues. Samples for schools are too small, for 
example, to permit HLM analyses despite the hierarchical structure of the surveys. 



5 David R. Mandel, “Teacher Education, Training and Staff Development: Implications for National Surveys” (paper 
prepared for the Conference on Future Directions of NCES Data Collection: Some Possible Directions, November 
1995), 15. 

6 David P. Baker, “Towards an Organizational Database on America’s Schools: A Proposal for the Future of SASS, 
with Comments on School Reform, Governance, and Finance” (paper commissioned for NCES seminar series on the 
1998-99 Schools and Staffing Survey), 4. 



Implications for Survey Depth 

1) NCES should take advantage of the fact that SASS, because of its linked-survey design, is 
uniquely positioned to monitor the extent to which various types of reforms are present in schools 
and in the classroom. SASS is the best vehicle NCES has for following the implementation of 
school reforms and for determining whether or not state-initiated reforms have filtered down to 
the district, school, and classroom. States can develop wonderful curriculum frameworks, for 
example, but if teachers do not actually use them, we cannot expect to see the effects on student 
learning. SASS can ask teachers questions about instructional practices, and we can use that 
information to examine implementation patterns by school and district characteristics. 

2) More information is needed to describe what goes on at the classroom level. In recent 
years, there has been increasing recognition that the quality of resources has to be measured at 
the classroom level. Teacher quality and opportunities to learn, not just district or school 
spending, which have traditionally been used as indicators of quality, are key to educational 
success, and therefore their distribution across schools and states is of major national interest. The 
implication for SASS is that more information about what is going on in the classroom is needed. 



Of the current SASS participants, teachers are the best source of information on their 
qualifications and on what is going on in the classroom. They can be asked, for example, what 
professional development activities they have participated in, if they are using a state curriculum 
framework, what assessment practices they are using, and how technology is used in the 
classroom. Self-report data, even with such straightforward items such as degrees earned and 
dates, pose problems, 7 but at the national level, is the best we can do. On a smaller scale, self- 
report data can be combined with other types of data collection such as video or case studies. 



Data Users and Uses 

SASS has been targeted to a wide audience, including policymakers at all levels, researchers, 
and the general public, and some of their data needs differ. Researchers, while interested in 
national data, typically do not need the large sample sizes required to support state-level 
estimates. Education administrators, on the other hand, want to know about variation across states 
as they make decisions about funding, reform initiatives, and staffing. 8 Researchers are typically 
trying to establish links between educational structures and practices and student outcomes, while 
administrators and policymakers are more frequently interested in monitoring the implementation 
of proven or desired policies and practices. 

From a survey depth perspective, the important question is whether the state- and association- 
level data are widely enough used to warrant continuing to produce estimates at this level. It is 
difficult to know the extent to which state data are used. NCES has published some state-by-state 
SASS data in the Statistical Profiles and in SASS by State, produced jointly with the Council of 



7 Bradford Chaney, The Accuracy of Teachers’ Self Reports on Their Postsecondary Education (Washington, D.C.: 
U.S. Department of Education, National Center for Education Statistics, Working Paper 94-04, July 1994), 8-12. 

8 U.S. Department of Education, National Center for Education Statistics, SASS by State, NCES 94-343 (Washington, 
D.C.: June 1994), 1. 



Chief State School Officers and Horizon Research, as well as the Digest and the Condition of 
Education, but it is reasonably safe to say that these publications do not provide everything a 
state could need. 

For state-level data to be really useful, the states need to have the capability to analyze SASS 
data themselves so they can tailor comparisons to their specific needs. The widespread availability 
of personal computers and appropriate software and NCES’s practice of providing large data sets 
on CD have made this increasingly feasible. A number of states have obtained the data 
themselves or requested analyses from the National Data Resources Center (NDRC). The data 
would be even more accessible if NCES were to create a Data Analysis System (DAS) for SASS 
such as those developed by Dennis Carroll for some of the other NCES databases. The DASs 
have allowed users with very limited computing capacity and technical knowledge to create their 
own cross-tabulations. 

The important point here is that rapid advances in technology are opening exciting possibilities 
for innovations in data collection, management, and distribution, and the movement is toward a 
closer connection between data producers and data users. 9 Statistics Canada recently held a 
symposium entitled From Data to Information that explored topics such as the role of the 
customer in “coproducing information,” integrating data collection and analysis, advances in 
analytic techniques of customers, and the privacy and confidentiality issues associated with new 
customer/supplier partnerships. 10 These trends suggest more, rather than less, demand for and 
use of state and other small area data in the future and a need for NCES to pay close attention 
to how SASS data are being or might be used. In the past, users of NCES data have been fairly 
well known to NCES staff, but as use becomes more dispersed, NCES may have to actively seek 
out the users to find out who is using what data and for what purpose. 

This discussion of policy issues and users of the data provides the context within which to 
proceed with the discussion of survey depth. In the rest of this paper, I return to the questions 
posed at the beginning: 1) At what level should estimates be provided? 2) From whom should 
data be collected? and 3) How much data can we collect from each respondent? In each case the 
issues are discussed first, followed by specific recommendations. 



SURVEY DEPTH 

1) At What Level Should Estimates Be Provided? 

Issues 

The 1993-94 SASS sample design provides estimates at both the national and state levels for 
public schools and teachers, and at the national and affiliation levels for private schools and 



9 Two papers presented at the conference on Future NCES Data Collection: Some Possible Directions address this 
area. See Fritz Scheuren, “Opportunities for Making More Effective Use of Administrative Records in Surveys of 
Elementary, Secondary, and Postsecondary Education,” and Glynn Ligon, “New Developments in Technology: 
Implications for Collecting, Storing, Retrieving, and Disseminating National Data for Education.” 

10 Reported in Amstat News, 227 (January 1996): 13. 



teachers. State/elementary and state/secondary estimates are also provided. Estimates of public 
and private libraries and librarians are provided at the national level. National estimates of public 
and private school students are provided by grade level and urbanicity. To improve the accuracy 
of estimates for certain sampling units that were relatively small in number but important for 
policy reasons, SASS oversampled schools in which more than 19.5 percent of the students were 
American Indian/Alaskan Native and included all Bureau of Indian Affairs (BIA) Schools, and 
oversampled bilingual/ESL, Asian/Pacific Islander, and American Indian/Alaskan Native 
teachers." 

The level at which estimates are provided has implications for cost, response rates, and the 
precision of estimates. The implications of each of these is discussed in turn. 

Cost. The decision to provide state- and affiliation-level estimates has implications for sample 
size and therefore a direct impact on the cost of administering the survey. To provide estimates 
at this level, the 1993-94 SASS included approximately 13,000 schools and 68,000 teachers. 
NCES could substantially reduce the sample size and therefore the cost of the survey if SASS 
provided only national estimates. The reduction in sample size could free up resources to enhance 
the survey in other ways, such as covering more types of school staff, conducting the survey 
more often, or increasing the amount of information collected in each survey. The important 
question, then, is: Are state- and affiliation-level data useful, or should the sample size be reduced 
and the resources reallocated? Are there better ways to spend the resources allocated to SASS 
than having such a large sample? 

Response rates. Survey depth may also affect response rates, although the precise impact, 
particularly of marginal changes, is not easy to predict or measure. State- and affiliation-level 
estimates might contribute to reduced response rates because the larger the sample, the greater 
the probability that a given respondent will be surveyed more than once over time. Respondents 
may view being asked to participate repeatedly as a significant burden and be more likely to 
decline to participate a second or third time. 

Providing state- and affiliation-level estimates, requiring a larger sample, could be jeopardizing 
district response rates. In the 1991 LEA pretest, response rates were significantly lower for 
overlap compared to nonoverlap LEAs (84 percent compared with 95 percent), suggesting a 
reluctance to participate repeatedly. 12 This problem could worsen over time as the amount of 
overlap increases with repeated SASS administrations, although the shift to a 5-year as opposed 
to 3-year interval between surveys will undoubtedly lessen the problem. Reducing the number 
of questions asked of districts (by dropping them or shifting them to the school questionnaire) 
might also improve response rates for overlap schools. 

Repeated participation does not seem to have had a negative effect on the school response 
rate. In the 1993-94 SASS administration, the response rate for overlap and nonoverlap schools 
was about the same among public schools (92 percent). Among private schools, overlap schools 



" This is somewhat of an oversimplification of the sample design. For more detail see U.S. Department of Education, 
National Center for Education Statistics and U.S. Bureau of the Census, 1993-94 Schools and Staffing Survey: 
Sample Design and Estimation (Washington, D.C.: forthcoming). 

12 NCES, 1993-94 Schools and Staffing Survey: Sample Design and Estimation, 39. 



actually had a higher response rate than nonoverlap schools (83 percent compared with 88 
percent). 13 

Although there is no evidence to prove this, it seems reasonable to hypothesize that providing 
affiliation-level estimates would affect the private school response rate positively rather than 
negatively. Private schools are not a “system” and typically have less interest than public schools 
in making intra-sector comparisons. The affiliation-level estimates allow private schools to define 
a more relevant comparison group than all private schools. Montessori schools can compare 
themselves to other Montessori schools, for example, and Catholic schools to other Catholic 
schools or to other religious schools. Without the affiliation-level estimates, private school 
associations might not be willing to endorse the survey. The response rate for private schools is 
already lower than for public schools (83 percent compared with 92 percent in 1993-94). 14 
Without association endorsements, the response rate might be even lower. 

Precision of estimates. Finally, survey depth has implications for the precision of estimates. 
The optimum sample design for national estimates is different from the optimum sample design 
for state-by- state comparisons. Schools and teachers should be sampled in proportion to their 
numbers in each stratum for national estimates, but equally among states for the best state 
comparisons. Since both types of comparisons have been important, the sample has compromised 
on these goals. 15 The precision of the national estimates could be improved by abandoning state- 
level estimates. 

The oversampling of certain types of teachers does not have cost implications because the 
oversampling does not change the total sample size. Larger samples of bilingual/ESL, Asian, and 
American Indian/Alaskan Native teachers are obtained by sorting the teacher lists so that adequate 
numbers of these types of teachers can be selected. However, oversampling does improve the 
precision of the estimates for these groups. 



Recommendations 

a) State- and affiliation-level estimates 

I believe that a strong case can be made for continuing to support state- and affiliation-level 
estimates given the purpose of SASS, the nature of emerging policy issues, and the context in 
which school reform initiatives are taking place. First, we need state-by-state data to describe 
accurately the education enterprise in the United States. Education is primarily a state and local 
responsibility and the current trend is away from federal involvement and uniformity at the 
national level, not toward it. Many important aspects of schooling are typically controlled or 
heavily influenced by state policy, such as programs offered, curriculum, graduation requirements, 
the number of days per year and hours per day that schools must be in session, pupil/teacher 
ratios, expenditures per student, teacher certification requirements, and salary schedules, to name 



13 NCES, 1993-94 Schools and Staffing Survey: Sample Design and Estimation , 39—40. 

14 NCES, 1993-94 Schools and Staffing Survey: Sample Design and Estimation , 4. 

15 U.S. Department ofEducation, National Center for Education Statistics, 1988 Schools and Staffing Survey Sample 
Design and Estimation, NCES 91-127 (Washington, D.C.: May 1991), 9-10. 



a few of the most important ones. Key features of the education system cannot be captured 
without state-level data. 

Second, the major current school reform efforts are being implemented at a sub-national level, 
even those initiatives originating at the federal level. The federal School-to-Work Opportunities 
Act, for example, has given the states grants and provided them a great deal of freedom to design 
their own programs. There is much more talk in Washington about block grants than about new 
federal programs. The reality is that most of the major reform efforts are being initiated at the 
state level. In the area of curriculum standards, for example, states have moved with varying 
strategies and at different speeds, with differences across states as noteworthy as the 
similarities . 16 To monitor reform, a state perspective is needed . 17 

Third, state-level data provide state policymakers with information useful for making state 
policy, such as scheduled salaries. States can generate some of this information themselves 
through their own databases, but SASS allows comparisons with other states, which is often hard 
to do when data do not come from a common source. 

Fourth, providing state-level data may build support for NCES data collections with federal 
legislators because their constituents will be happy. In a similar vein, providing affiliation-level 
data will make the private school associations happy, thus encouraging them to endorse the 
survey, which, as suggested above, is likely to improve the private school and teacher response 
rates. 

Finally, it would be a shame to eliminate state-level estimates just when the technologies being 
developed for data collection, management, and dissemination are leading in the direction of 
increased usefulness of nationally collected data at subnational levels. 

The major argument against continuing to support state- and affiliation-level estimates is cost. 
Cost savings could be a strong argument if you had better ways to use the funds. The major 
pressure right now seems to be how to get more information from teachers. Eliminating state- 
and affiliation-level estimates would not do much to accomplish this goal because the chief 
difficulty is the response burden on individual teachers. 

b) Oversampled groups 

Continuing to oversample schools with large American Indian/Alaskan Native student 
populations and to oversample bilingual/ESL, Asian/Pacific Islander, and American 
Indian/Alaskan Native teachers also seems appropriate. The issues that prompted the oversampling 
in the first place have not gone away. The only question would be whether other groups have 
greater priority, and none is apparent. 



16 David K. Cohen, “What is the System in Systemic Reform?” Educational Researcher, 24.9 (December 1995): 
11 - 12 . 

17 Jennings and Stark recommend that NCES consider studying education reform on a state-by-state basis and issue 
annual reports on state activities. See John F. Jennings and Diane Stark, “Tracking Education Reform: Implications 
for Collecting National Data Through 2010.” 



The American Indian/Alaskan Native student population is relatively small (about 1 percent 
of the total student population) and therefore would not be well enough represented in a national 
survey of schools and staffing to permit reliable generalizations about their characteristics. Yet 
there are policy concerns about the condition of education for these students and some systematic 
differences in the characteristics of these schools and staff compared with other schools and staff, 
and these bear monitoring. 18 

In the United States as a whole, 14 percent of all children aged 5 to 17 years spoke a language 
other than English at home in 1990, and 5 percent had difficulty speaking English. The 
percentage of school-age children with difficulty speaking English increased by 27 percent 
between 1980 and 1990. 19 It will continue to be important to monitor the adequacy of the supply 
of teachers to help these students. 

Asian and American Indian/Alaskan Native teachers made up such a small percentage of the 
total teacher population (1.1 percent and 0.7 percent, respectively, in 1993-94), 20 that it would 
be difficult to study them without oversampling them. Accurate data on minority teachers is 
important because the adequacy of the supply of minority teachers and their distribution among 
different types of schools remain important issues. 



From Whom Should Data be Collected? 

Issues 

One of the major strengths of SASS is the integrated survey design that links schools, 
teachers, districts (for public schools) and, as of 1993-94, libraries, librarians, and students. 
Questionnaires are completed by districts, schools, principals, teachers, and librarians. Two issues 
are important to consider here. One is whether this is the correct structure. Are we getting all the 
information we want, or should we include other staff such as vice principals, department heads, 
or counselors? 

The other issue is whether we are asking the right questions of the right people. There are two 
reasons we might want to change. One is to improve accuracy. For example, until the 1993-94 
administration of SASS, information on public school teacher benefits was collected from the 
district, which was asked which of a list of benefits were offered to teachers. Beginning in 
1993-94, questions on benefits were shifted to teachers, who were asked what benefits they 
received, which was much more useful because it provided more accurate information and 
allowed direct comparisons with benefits received by principals. The second reason we might 
want to change the source of information is to reduce the response burden imposed on particular 
types of respondents. For example, if some questions currently on the district questionnaire could 
be shifted to the school questionnaire, the response burden for the districts could be reduced. 



18 U.S. Department of Education, National Center for Education Statistics, Characteristics of American Indian and 
Alaskan Native Education, NCES 95-735 (Washington, D.C.: April 1995), 1. 

19 U.S. Department of Education, National Center for Education Statistics, Condition of Education 1994, NCES 
94-149 (Washington, D.C.: August 1994), 130. 

20 NCES, Schools and Staffing in the United States: A Statistical Profile, 1993-94. 



Recommendations 



Schools employ a wide variety of types of staff, so collecting information from principals and 
teachers necessarily provides only a partial picture of school conditions. Other staff who could 
provide information on school conditions include: vice principals, department heads, counselors, 
school superintendents. The major problem with including these types of staff is that their roles 
vary widely from school to school, giving them quite different perspectives. School 
superintendents would be a useful source of information on both state an district policies, but 
probably no better than whoever fills out the district questionnaire. It might be interesting to learn 
about superintendents’ backgrounds, education, and career paths, but this would not be a high 
priority given the ambitious goals already outlined for SASS. 

If extending the survey depth to collect more information about what goes on in the classroom 
is considered desirable, the question then arises as to who should provide the information, 
teachers or students. The current TFS survey contains a considerable amount of information on 
teaching methods collected from teachers. Judgments on the usefulness of this information will 
have to wait until the data on classroom practices are evaluated. Self-reported data on this topic 
may prove valuable, but could prove to have significant limitations. It remains to be seen. The 
issue of whether to include students in SASS is dealt with in another paper and so is not 
addressed here . 21 

Over the three administrations, NCES has switched the sources for some data (such as 
teachers’ benefits) to improve quality. Overall, it appears that the right questions are now being 
asked of the right individuals to maximize accuracy. However, since the district questionnaire 
takes the longest to complete, it is worth considering whether the burden on the districts could 
be reduced by shifting some questions to the schools. Some district policies would be known at 
the school level, such as the number of days in school year or whether or not the district had a 
choice program (although schools would not know how many students participated). Collecting 
such data from schools could lead to the problem of reconciling data from multiple schools in 
the district. Also it could irritate the school personnel filling in the questionnaire, who might 
wonder why the questions were being asked of them rather than the district. Some information 
on district policies would only be available reliably from the districts, such as salary schedules, 
incentive pay policies, hiring criteria, and policies on retraining. On balance, there does not seem 
to be much prospect for shifting much of the response burden away from the districts. The most 
burdensome questions are those that require looking up numbers, such as student enrollments, and 
the district is the only source of this information. 

As a final point, it is worth keeping in mind that the appropriate level to collect data may 
change in the future with advances in data collection technology. For example, information on 
enrollments now collected from the schools may be more easily obtainable from a district or even 
state database. 



21 Phillip Kaufman, “Student-Level Data: If, When, and Where,” 
the 1998-99 Schools and Staffing Survey). 
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How Much Data Can We Collect? 



Issues 

Once a survey has been developed, the marginal cost of fielding an additional question is 
relatively low. It is very tempting to keep adding questions, but the response burden cannot be 
ignored. Estimated times for completing the surveys (printed on the questionnaire) were as 
follows: LEAs, 2 hours; public schools, 1 hour; private schools, 2 hours; and principals, 30 
minutes. A time for teachers was not estimated, but the average time reported by public school 
teachers in 1993-94 was 36 minutes. The overall burden on the principal may be considerably 
greater than 30 minutes depending on how much of the school questionnaire the principal has to 
complete personally. In a small school with no other administrators, the burden for responding 
to the school questionnaire and developing the teacher list probably falls to the principal. 

A twist on this issue is that the types of questions may be more relevant than the length of 
time it takes to respond. Respondents may not object to completing a long questionnaire in which 
the questions are easy to answer, but may balk at a short questionnaire that requires them to look 
up numbers or that includes questions that are puzzling or difficult to answer. Thus, NCES 
should pay close attention to the types of questions asked as well as how long they take to 
answer. 

To date, overall response rates have been good, suggesting that the burden is not yet 
excessive. 22 However, the response rate to the initial mail survey (as opposed to the telephone 
followup) was considerably lower, suggesting that there may already be too many questions for 
some. Although the response rate has been increased by telephone followup, reinterviews have 
shown that the responses obtained through the mail survey were more reliable. 23 



Recommendations 

The major increase in response burden for new data collection to address emerging issues will 
fall on the teacher. For example, including all the information asked on the TFS in the regular 
SASS teacher questionnaire would greatly increase the response burden for teachers. A variety 
of options exist to address this, including the following: 

1) Assume this is not a problem, and add to the questionnaire. One could point to the high 
response rates attained in previous administrations of SASS and argue that more questions could 
be added without reducing the response rate. This is an empirical question, at least for now, and 
could be answered with a field test. 



22 In the public sector the weighted response rates in 1993-94 were 93.9 percent for districts, 96.6 for principals, 92.3 
percent for schools, and 88.2 percent for teachers. In the private sector they were 87.6 percent for principals, 83.2 
percent for schools, and 80.2 percent for teachers. See NCES, 1993-94 Schools and Staffing Survey: Sample Design 
and Estimation , 4. 

23 John M. Bushery, Daniel Royce, and Daniel Kasprzyk, “The Schools and Staffing Survey: How Reinterview 
Measures Data Quality,” in U.S. Department of Education, National Center for Education Statistics, Schools and 
Staffing Survey : Papers Presented at Meetings of the American Statistical Association, Working Paper No. 94-01 
(Washington, D.C.: July 1994), 7. 



2) Try to maintain the amount of time that it takes to complete the questionnaire while 
increasing the number of questions. Changes in technology may help make this possible. At some 
point in the not-too-distant future, most teachers will have access to and be able to use computers. 
It may thus be not only feasible, but very efficient to provide the questionnaires to teachers in 
electronic form, at least as an option. A teacher might be able to move considerably more quickly 
and easily through the questionnaire if he or she did not have to pay attention to skip patterns. 
Electronic data collection from teachers and schools is already being investigated. 2 '* This could 
reduce the cost of data collection per teacher (making larger samples feasible) and improve 
accuracy (by eliminating skip pattern errors, for example). To what extent it would reduce 
response time enough to permit asking a lot more questions is largely unknown, but bears 
investigating. 

Also, as indicated above, it is not only the number of questions but also their difficulty that 
affects response rate. If the questions are interesting to teachers and easy to answer, they may not 
take much longer. 

3) Eliminate some of the current questions. It is very difficult to identify items that could be 
dropped from the teacher questionnaire, but a couple of suggestions are made here. Among the 
least useful seems to be the detail on what teachers were doing before they started teaching at that 
school (Q. 6-11). When they first taught, how and many years of total teaching experience they 
had could be sufficient. The detail on the number of courses ever taken in various fields may be 
unnecessary. This is a lot of work to figure out and may not be very meaningful or accurate 
except for recent graduates. This may be the type of question that causes teachers to give up. 
They could answer a lot of questions in the time it takes to dig up all that information. The 
questions on majors and minors may be enough. 

4) Don ’t ask all teachers all questions. This would require increasing the total sample size to 
continue state- and affiliation-level estimates for all data items. The division of questions would 
have to be done very carefully to make sure that you did not split data among subsamples that 
you want to examine together. 

5) Cycle some questions. For example, on one survey you might ask about instructional 
practices and on the next perceptions of school problems or decision making. This means that 
some questions would only be asked every 10 years. In reality this might not be a serious loss. 
Comparisons of data from 1987, 1991 and 1994 show considerable stability in many areas. 
However, policymakers and the general public increasingly expect up-to-the-minute data; 1 0-year- 
old data have little credibility. 

6) Use TFS to collect data for which national estimates are sufficient and/or to field test new 
data. In 1994-95, the TFS is being used to question teachers extensively about their instructional 
practices. We don’t know yet what the quality of these data will be, but if it seems worthwhile 
to collect similar information again, the TFS may be an appropriate permanent home simply 



24 Daniel Kasprzyk, “The Schools and Staffing Survey: Research Issues,” in U.S. Department of Education, National 
Center for Education Statistics, Schools and Staffing Survey : Papers Presented at Meetings of the American Statistical 
Association, Working Paper No. 94-01 (Washington, D.C.: July 1994), 3. 



because there may be too many questions to add to the regular teacher survey. The TFS is a good 
testing ground for new subject areas because the sample is small yet nationally representative. 

7) Use the SASS sample as a framework for more limited studies. Metcalf, for example, has 
explored the feasibility of incorporating experimental designs into NCES data collection 
methodologies. 25 If everything we are interested in cannot be addressed in a national data 
collection, it might still be possible to use SASS as a sampling framework for additional, more 
specialized studies. 

More information is needed to know which of these options (or combinations of options) are 
most promising. In particular, we need to know how successful the TFS data collection on 
instructional practices turns out to be and what the survey scope will be. 



CONCLUSION 

My major points can be summarized briefly as follows: 

1) SASS has been an extremely valuable survey, and its current structure of an integrated set 
of linked surveys is useful and appropriate to meet the purposes of the survey and to provide 
information on emerging policy issues. 

2) SASS is the best mechanism NCES has for monitoring the implementation and diffusion 
of many current school reform initiatives. 

3) Given the current focus on measuring quality at the classroom level, SASS needs to reach 
more extensively into the classroom. This means detailed information on teacher training and 
professional development (a direction in which the 1993-94 SASS has already moved) and on 
instructional practices (beginning to be addressed through the TFS). 

4) State- and affiliation-level estimates should be continued, as should the current 
oversampling of schools with large American Indian/Alaskan Native student populations and 
oversampling of bilingual/ESL, Asian, and American Indian/Alaskan teachers. 

5) No additional types of respondents should be added to SASS. 

6) Because of the interest in focusing on the classroom, the increased response burden will 
fall mainly on teachers. Simply adding the TFS questions to the next full teacher survey may not 
be realistic. A variety of options exist to deal with this problem, but the results of the TFS survey 
and decisions on survey scope are needed before recommendations can be made. 



25 Charles E. Metcalf, “Incorporating Experimental Designs into New NCES Data Collection Methodologies,” paper 
prepared for the Conference on Future Directions of NCES Data Collection: Some Possible Directions, November 
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